Driven by a desire to improve management of big data, data processing power and data center content is being moved to the edge of a network instead of being held in a cloud or at a central data warehouse. This move to edge computing is advantageous, for example, in Industrial Internet of Things (IIoT) applications such as power production, smart traffic lights and manufacturing. Edge devices capture streaming data that can be used, for example, to prevent a part from failing, reroute traffic, optimize production and prevent product defects. With ever increasing unstructured data (e.g., video, sensor, etc.), existing data center based processing solutions cannot perform real time analytics and respond back to users due to high latency and significant bandwidth associated with large data sets.
Conventional networks, which feed data from devices for transactions to a central storage hub (the old data warehouse model), cannot keep up with the data volume and velocity created by Internet of Things (IoT) devices. The data warehouse model also cannot meet the low latency response times that users demand. Sending data to the cloud for analysis also poses a risk of data bottlenecks as well as security concerns. New business models need data analytics in a minute or less; in some cases in less than a second. The problem with data congestion will only get worse as IoT applications and IoT devices continue to proliferate. Security cameras, phones, machine sensors, thermostats, cars and televisions are just a few of the items in daily use that create data that can be mined and analyzed. Add the data created at retail stores, manufacturing plants, financial institutions, oil and gas drilling platforms, pipelines, and processing plants to the above data, and it is not hard to foresee a deluge of streaming and IoT sensor data can and will very quickly overwhelm existing conventional data analytics tools.
It would be desirable to implement hierarchical caching and analytics.